We are using a two-component hurdle model: first, the model predicts whether a disease will be present (binary), and if present, it predicts the case count (integer). Here we compare the results of a boosted tree model to our baseline model.

Disease Status

disease status confusion matrix
.metric desc model full_model
accuracy proportion of the data that are predicted correctly baseline 0.85
xgboost 0.96
kap similar measure to accuracy(), but is normalized by the accuracy that would be expected by chance alone and is very useful when one or more classes have large frequency distributions. baseline 0.45
xgboost 0.88
sens the proportion of positive results out of the number of samples which were actually positive. baseline 0.99
xgboost 0.98
spec the proportion of negative results out of the number of samples which were actually negative baseline 0.36
xgboost 0.88
disease status confusion matrix by taxa
.metric model birds buffaloes camelidae cats cattle cervidae dogs equidae hares/rabbits sheep/goats swine
accuracy baseline 0.85 0.77 0.780 0.76 0.86 0.730 0.78 0.90 0.85 0.86 0.87
xgboost 0.95 0.96 0.960 0.96 0.95 0.960 0.95 0.97 0.96 0.96 0.96
kap baseline 0.42 0.21 0.130 0.36 0.57 0.084 0.49 0.42 0.19 0.46 0.43
xgboost 0.85 0.91 0.890 0.92 0.88 0.910 0.89 0.87 0.86 0.88 0.87
sens baseline 0.98 1.00 1.000 1.00 0.99 1.000 0.99 0.99 0.99 0.99 0.99
xgboost 0.98 0.98 0.980 0.98 0.98 0.980 0.97 0.99 0.98 0.98 0.98
spec baseline 0.34 0.15 0.094 0.30 0.50 0.061 0.45 0.31 0.14 0.37 0.33
xgboost 0.86 0.92 0.910 0.94 0.89 0.920 0.91 0.85 0.85 0.89 0.86
disease status confusion matrix by continent
.metric model Africa Americas Asia Europe NA Oceania
accuracy baseline 0.83 0.82 0.85 0.87 0.94 0.930
xgboost 0.95 0.96 0.96 0.95 NA 0.990
kap baseline 0.47 0.39 0.47 0.47 0.51 0.094
xgboost 0.88 0.91 0.89 0.83 NA 0.890
sens baseline 0.99 0.99 0.99 0.99 0.99 1.000
xgboost 0.98 0.98 0.98 0.98 NA 1.000
spec baseline 0.39 0.31 0.38 0.38 0.42 0.054
xgboost 0.89 0.92 0.89 0.83 NA 0.870
disease status direction change confusion matrix
.metric desc model full_model
accuracy proportion of the data that are predicted correctly baseline 0.850
xgboost 0.960
kap similar measure to accuracy(), but is normalized by the accuracy that would be expected by chance alone and is very useful when one or more classes have large frequency distributions. baseline 0.046
xgboost 0.500
sens the proportion of positive results out of the number of samples which were actually positive. baseline 0.460
xgboost 0.560
spec the proportion of negative results out of the number of samples which were actually negative baseline 0.680
xgboost 0.790
Note there are baseline cases where disease status is positive but cases are NA, which are imputed in the model as 0.
disease status direction change confusion matrix by taxa
.metric model birds buffaloes camelidae cats cattle cervidae dogs equidae hares/rabbits sheep/goats swine
accuracy baseline 0.850 0.770 0.780 0.760 0.860 0.73 0.7800 0.900 0.850 0.860 0.870
xgboost 0.950 0.960 0.960 0.960 0.950 0.96 0.9500 0.970 0.960 0.960 0.960
kap baseline 0.071 0.017 0.048 -0.028 0.042 -0.02 -0.0014 0.063 0.026 0.044 0.065
xgboost 0.420 0.640 0.640 0.720 0.460 0.72 0.5900 0.470 0.500 0.510 0.420
sens baseline 0.450 0.560 0.580 0.610 0.430 0.59 0.4700 0.480 0.420 0.460 0.500
xgboost 0.520 0.610 0.600 0.560 0.540 0.59 0.5700 0.570 0.560 0.550 0.520
spec baseline 0.690 0.650 0.670 0.630 0.670 0.62 0.6500 0.690 0.670 0.680 0.690
xgboost 0.760 0.840 0.840 0.880 0.780 0.88 0.8300 0.790 0.790 0.800 0.770
disease status direction change confusion matrix by continent
.metric model Africa Americas Asia Europe NA Oceania
accuracy baseline 0.830 0.820 0.850 0.870 0.940 0.930
xgboost 0.950 0.960 0.960 0.950 NA 0.990
kap baseline 0.024 0.014 0.054 0.088 -0.019 0.065
xgboost 0.520 0.550 0.540 0.400 NA 0.460
sens baseline 0.450 0.450 0.470 0.470 0.330 0.590
xgboost 0.550 0.560 0.590 0.520 NA 0.540
spec baseline 0.660 0.660 0.680 0.690 0.660 0.710
xgboost 0.800 0.810 0.810 0.760 NA 0.780

Cases

Here we evaluate the subset of the training data with positive case counts

cases model stats

## # A tibble: 6 x 4
##   model    .metric .estimator  .estimate
##   <chr>    <chr>   <chr>           <dbl>
## 1 baseline rmse    standard   105536.   
## 2 xgboost  rmse    standard   285638.   
## 3 baseline rsq     standard        0.907
## 4 xgboost  rsq     standard        0.540
## 5 baseline mae     standard     1234.   
## 6 xgboost  mae     standard     2231.
cases residuals
cases residuals by taxa
cases residuals by continent